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ABSTRACT 

Evaluated was a 3-year gradpat^ program to train 
teachers of emotionally disturbed (ED) children. The program^s major 
goals were the acquisition of basic knowledge in special education 
and child development, the attainment of skills for professional 
teaching competence, and the development of deepened sensitivity to 
oneself and others. The training sequence involved an intensive 
learning experience (seminars and internships) and practicums 
experience in different activities (such as classroom teaching) at a 
therapeutic school for ED pupils. Before and after each academic 
year, three groups of eight students each were administered test 
batteries (measuring competencies, attitudes toward education, and 
interpersonality characteristics) along with two self •evaluation 
instruments. Results comparing pre* and post-program attitudes showed 
that trainees reported they felt less disturbed by overly aggressive 
behavior of children, increased their skill in decoding affective 
statements, tended to see others as less lovable and less competent, 
and checked a greater number of favorable adjectives when rating 
themselves. Follow-up interviews indicated that graduates felt the 
program provided the most effective training for those areas 
(personal sensitivity and behavior management) which they also 
considered most relevant. (Tables and figures are provided,) (SB) 
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OVERVIEW 



For the past four years we have been involved in develop 
ing an alternative training model for teachers who want to 
learn -skills in assessing-, in .stnucting., and managing pupils 
with severe learning and behavioral problems. Based on 
specific competencies., a prolonged practicum experience and 
'close supervision, the program was developed at Hillcrest 
Children's Center, a therapeutic school for .emotionally dis- 
turbed pupils from the Greater Washington Area. While this 
model solved many of the nagging problems of a University- 
Based Training Model, it also created many new ones. In an 
attempt to evaluate the strengths and weaknesses of our model, 
a three-year analysis is presented, for your evaluation. The 
report includes a Description of the Program, and a Program 
Evaluation including our graduates, their analysis of our 
Competency Measures, and our analysis of their personality 

characteristics . 

During this period we have developed additional mater- 
ials that are available upon request such as; (1) The Video- 
Tape Program, (2) The Screening Procedures and Process for 
Applicants, (3) The Apprenticeship Model of Training. 
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In September, 1972, the program was moved from Hill- 
crest, a private facility, to The Rose School, a public 
facility of The District of Columbia Department of Human 
Resources and The District of Columbia Public Schools Depart- 
ment of Special Education. This change keeps the program 
in the eye of the educational storm regarding the public 
school's responsibility, and programs for exceptional pupils. 
While this position is uncomfortable at times, we feel this 
is where University Training Programs belong. 

Nicholas J.. Long* 



8 



erJc 



DESCRIPTION OF PROGRAM 

For several years, The American University was inter- 
ested in developing a graduate, program in the area of teacher 
training for emotionally disturbed children. The University 
determined that the plans for developing a new major program 
should properly include: direct influence in and avail- 
ability of a special education center that would function 
as a Training Center, the hiring of a recognized authority 
,in the field, and a training program that would be experi- 
mental, imaginative, and comprehensive. 

The university conditions were fulfilled by the fol- 
lowing measures. In 1968 the Board of Directors of Hillcrest 
Children's Center and American University's Department of 
Education agreed to an arrangement in which the pre-school 
and elementary school programs of the Psychoeducational 
Institute at Hillcrest would become the Training Center for 
American University. Dr. Nicholas J. Long, Director of the 
Psychoeducational Institute, was appointed Professor of 
Education at American University and designated as Training 
Director for the program. The psychoeducational staff of 
Hillcrest became the faculty for a Center for Special Educa- 
tion in the Area of Emotionally Disturbed, with each teaching 

3 
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staff member designated as Adjunct Professor or Lecturer 
of the University. Because of the uniqueness of this 
agreement, in which University faculty was also responsible 
for the on-going educational program, an innovative and ex- 
perimental, teacher training program was developed, weaving 
together as a unified experience academic study and daily 
practice teaching. , - 

The prograia was funded by a demonstration grant from 
the U.S. Office of Education, Bureau of Handicapped for a 
three-year period: June, 1969 through May, 1972, To date, 
three groups of eight students have graduated from the 
program and received Master^ s degrees in Education, with 
■concentration in Special Education. 

Program Objectives 

The A.U.-Hillcrest teacher training program had two 
basic premises. The first was that facilities such as 
Hillcrest could become unique training centers for graduate 
students in special education through university affiliation. 
Academic study and practicum experience could be successfully 
integrated in such an affiliation. The second premise was 
that qualified college graduates with no undergraduate teacher 
training courses could become competent special educators 
of emotionally disturbed children in one year, in particular 
because of the constant combination of practicum and theory. 



The objective for graduates of this program was 

•••• ^ 

that they become professionally competent to develop, main- 
tain, and evaluate a healthy educational program for emo- 
tionally disturbed children. To accomplish this, academic 
pursuit and practical experience were focused on seven major 
training .goals. These seven goals, outlined in detail below, 
covered three major areas of student development: (1) the 
acquisition of basic knowledge in special education and 
child development; (2) the acquisition of basic knowledge 
and skills for professional teaching competence (curriculum, 
psychoeUucational assessment, teaching methods, behctvior 
management); and (3) the development of deepened sensitivity 
to oneself and to others, which is stressed throughout the 
year as is the ability to use this awareness in teaching 
children and working with 'colleagues. 

Goal 1: To De^^lop an Awareness o f how Family , Social 
and Educational Forces Influence the Pupil^s Behavior and 
Adjustment in School , 

Sub--goal (a) . Ability to formulate and communicate 

concepts of how systems function, e.g., familial, 

cultural, educational. 

Sub--goal (b) . Ability to relate individual character- 
istics and behavior to group and system transactions. 
Sub-Roal (c) . Ability to relate individual character- 
istics and behavior to life style culture, with 

particular emphasis on urban Black. 

ijl. 
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Goal 2; To Develop Personal. 'Sensitivity 
Sub-goal (a) . Ability to comprehend and communicate 
effectively with others: to perceive accurately one's 
reactions to and effect upon others . 

Sub-goal (b) . Ability to accept and to freely express 
positive and negative emotions. 

Sub-goal (c) . Ability to promote mutual understand- 
ing of problems and the exchange of ideas among co- 
workers. 

Goal 3; To Develop Basic Knowledge of Concepts and 
Practices of Child Development and Special Education 

Sub- goal (a) . Appreciation of the influences of 
constitutional, maturational and environmental factors, 
and their interaction, on child development. 
Sub-goal (b) . Knowledge of alternative models of 
special education and their application. . • 

.4* 

Goal To Develop Basic Knowledge of Psychoe'ducational 
Assessments 

Sub-goal (a) . Knowledge of measurement theory, con- 
cepts and problems. 

Sub-goal (b) . Ability to create an optimal atmos- 
phere for assessment, including appreciation for both 
empathy and structure. 
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Sub-goal (c) .. Ability to,pbserve pbj. actively and 
to evaluate data meaningfully. 

Sub-goal (d) . Use of assessment information for psycho- 
educational, programming, planning, and follow-up. 

Goal 5: To Develop Basic Knowledge of Curriculum and 
Teaching Methods in Special Education 

Sub- goal (a) . Basic knowledge of subject matter and 
teaching methods in language arts, mathematics, science 
and social studies. 

Sub-goal (b) . Ability to present educational, mater- - 
ial and tasks at a level of reasonable challenge for 
each child in the classroom. 
, Sub-goal (c) . Ability to make curriculum and teaching 
techniques responsive and relevant to the needs of 
each child in the classroom. 

Goal 6: To Develop Basic Knowledge of Remedial 
Education . 

Sub-goal (a)_ . Familiarity with special curriculum ■ 
resources and activities for specific learning deficits 
Sub-goal (b) . Ability to evaluate learning outcomes 
in relation to progressive development of educational 
or teaching sequences. 

Sub-goal (c) . Ability to structure close teacher-pupil 
relationships in order to facilitate learning and self- 
fulfillment. ' 'I ;.! 
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Goal 7; To Develop a Basic Knowledge* of the Manage- 
ment of Behavior in Children^ 

Sub-goal (a) > Ability to establish and present 
realistic behavior standards (limits) in an educa- 
tional setting. 

Sub-goal (b) . Ability to identify, support > and promote 
appropriate group and individual behavior of pupils 
; in an educational setting. ' 

Sub-goal (c) . Ability to develop and use techniques 
of teacher intervention to protect the group and pupil 
from disruptive school behavior. 

Sub-goal (d) . Ability to educationally exploit in- 
appropriate classroom behavior to teach new skills 
for coping with interpersonal and curriculum tasks. 

Training Sequence 

Trainees in the A.U.-Hillcrest program were involved 
in an intensive learning experience for a full year. During 
1969-1970 and 1970-1971, the program actually took place in 
three different settings: The American University campus, 
Hillcrest Children's Center, and special classes in the 
Montgomery County (Md.) Public Schools ♦ An overview of the 
training sequence is presented in the following diagram. 
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1. Summer session at American University Campus. 

2. Six full weeks teaching in public school special classes . 

(3 weeks in September and 3 weeks in January). 

3. Seven and a half months at Hillcrest Children's Center 

teachijig on a full-time basis, including daily seminars, 

4. One week in the school and residence program on. a 2U-hour 

» «» 

basis • 

5. Approximately N^very six weeks the school closes in order 
for the totaisstaff and trainees to evaluate the 

training progr^. 

During academic 1971-1972, a three-week orientation 
program was held during September and public school placement 
consisted of four weeks in the District of Columbia, Prince 
George's and Montgomery Counties (Md.) school systems in 
January. 

Academic Program , 

During each year, a total of 13 courses were presented 
at A.U. and at Hillcrest. Thirty-seven credits were granted 
upon completion of the year: 29 credits for the courses, 
with 8 additional credits granted for the internship practicum< 
A listing of the courses and the credit. hours assigned to 
each follow: 

Section 1. American University Summer Session : 

a. Philosophy of Education 3 hours 

b. Advanced Educational Psychology t hours 

c. Research Methods in Education 3 hours : 
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Section 2. First Semester^ Seminars / " 

a. Theories and Methods of Urban Education 

b. Method of Studying Group Behavior 

c. Theories of Child Development 

d. Language Arts and Social Studies Cur- 
riculum 

e. Methods of Remediation in Special 
Education 

f . Methods of Managing Inappropriate 
Classroom Behavior 

Section 3. Second Semester Seminars 

a. Theories and Methods of Urban Education 

b. Methods of Group Dynamics in Special 
Education 

c. Theories and Practices in Special 
Education 

d. Psychoeducational Assessment 

e. Mathematics and Science Curriculum 
f • Methods of Remediation in Special 

Education 

g. Methods of Managing Inappropriate 
Classroom Behavior 

Section M-. Internship 

Trainees spent the sununer on the A.U. campus enrolled 
in three courses required for all cnadidates for the Master 



2 hours 
2 hours 
2 hours 

2 .hours 

2 hours 

2 hours 

(cont'd) 

2 hours 

2 hours 
2 hours 
2 hours 

(contM) 

(contM) 
8 hours 
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of Education degree. These courses were taught by University 
faculty other than Hillcrest staff. Philosophy of Education 
examined the rationale underlying the process, purposes and 
methodology of education. Advanced Educational Psychology 
focused on the psychological principles and current research 
studies related to education. Research Methodology reviewed 
research design with primary emphasis on understanding and 
evaluating educational research reports. 

Seminars taught at Hillcrest by Hillcrest staff were 
taken simultaneously with daily half-day sessions of student 
teaching under the constant supervision of master teachers. 
This created a constant two-way flow of theory and practice: 
theories presented in a seminar were often seen illustrated 
in the next day's session with the children. A brief des- 
cription of each seminar follows. 

Theories and Methods of Urban E ducation was designed 
to demonstrate how the total environment in which inner city 
children live affects their ability to learn, feel, and 
behave and how inner city schools in VJashington, , 
function. To facilitate these two goals, the course included 
bi-weekly field trips to appropriate community agencies, 
schools and neighborhoods/ 

Methods of Studying Group Behavior was designed along 
the lines of a process group. Under the careful direction 
of a trained group specialist, the trainees had an opportunity 

r-'' 1,8 
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to explore interpersonal and intrapersonal conununications 
and understand how groups function. These experiences 
were related to the role of the classroom teacher in special 
education. 

Theories of Child Development reviewed basic theoi?ies 
of child development including psychoanalytic, social learn- 
ing and cognitive-developmental theories. The course focused 
on such concepts as critical periods, stage and phase 
development, primary and secondary drives and reinforcement > 
hierarchy of motivational needs, defense mechanisms, develop- 
mental lines and self-mastery. Material on early childhood 
differences in constitution, personality and rearing prac- 
tices were covered and related to the students in the 
trainees' classroom. Additional didactic learning was 
obtained through seminars, conferences, and institutes 
offered at the Children's Hospital of the District of * Columbi 
particularly in the Department of Psychiatry. 

Language Arts and Social Studies Curriculum examined 
the relationship between the teacher and his curriculum, and 
the learner and his need system. The first semester concen- 
trated on language arts and social studies including the 
following aspects: recognition of the teaching of reading 
as a crucial educational responsibility, insight into the 
reading process in relation to principles of learning and 
child development, materials and techniques for specific 
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learning outcomes, the role of language in the child's 
life, and the interrelationships among various phases of 
language arts. 

Methods of Remediation reviewed remedial principles 
of teaching children with learning problems. The seminar 
covered basic phonetic approaches, construction and admin- 
istration of informal reading and developmental skill 
inventories 5 and teaching programs for skill development. 
Practicum experience included individual and group observa- 
tion and teaching, lesson planning, supervisory conferences, 
developing teaching resource file, and, during the second 
semester, tutoring three times per week. 

Managing Inappropriate Classroom Behavior reviewed 
the theories of management based upon the psychoanalytical, 
psychoeducational, behavioral, and educational models. 
Particuicir attention was given to understanding and develop- 
ing skills around behavioral management as well as life 
space intervi^ewing skills. In addition to readings and 
discussion, trainees discussed examples of behavioral crises 
from their supervised classroom experience. Many of these 
^'examples" were available on video-tapes. 

Methods of Group Dynamics in Social Education con- 
tinued to provide the trainees with an opportunity to explore 
the techniques of interpersonal and intrapersonal group 
communication and understanding. Continued efforts were made 
to relate these experiences to the teaching role. 
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Theories and Practices li) Special Education covered 
three phases during the semester: overview of the field 
of exceptional children, theories of re-education, and 
current research and writings in the field of special educa- 
tion. 

The Seminar in Psychoeducational Assessment was 
designed to develop an appreciation of the assessment pro- 
cess by which one attempts to understand a child 's> learning 
and behavior. The course examined critical issues and con- 
cepts in measurement of abilities and behavior, data inter- 
pretation, psychoeducational prograjnming and follow-up 
evaluation. Practicum experiences included observation 
and participation in classroom activities, group and indi- 
vidual testing, and psychoeducational case conferences. 

Math and Science Curriculum in Special Education 
replaced the language arts and social studies seminar during 
the second semester. This course covered basic material in 
math and science through readings, demonstrations, educa- 
tional games, and laboratory exercises. All material develop 
was implemented in the trainees* classroom whenever approp- 
riate. 

An additional non-credit seminar met monthly for 
discussion of books selected by staff members as particularly 
topical and relevant. For example, titles selected for this 
"Great Books Seminar" in 1970-71 were: 
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September: Hov; Children Fail aJid. How Children Learn (Holt) 
October: The Emotionally Disturbed Child in the Class- 
room (Hewett) 



November: 
February : 
March : 
April : 
May : 
June : 



Black Skin; V/hite Mask (Fanon) 
Controls from VJithin (Redl) 
The Other 23' Hours (Trieschman) 



The Magic Years (Fraiberg) 
Reality Therapy (Glasser) 

Mario The Magician (Mann), and The Magus (Fowles) 
The relationship between the seven program goals and 
the various seminars and courses is indicated in the follow- 
ing outline. 

SEMINARS/COURSES 



GOALS 

1. System Awareness^^ 

2. Personal Sensitivity 

3. Child Development S 

Special Education 



1. Urban Education 

2. Group Behavior 

1. Group Behavior * 

1. Child Development 

2. Special Education** 

3. Psychology of Education 
U. Philosophy of Education 



U. Psychoeducational Assess- 1. Assessment** 

ment 2. Research Methods 

5. Remedial Education 1. Remediation** 



*Once monthly: didactic on administration and didactic 
on family (case conference). 

**These 3 seminars had an associated practicum: Socio- 
metric techniques, testing and tutoring, respectively. 
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GOALS (contM) SEMINARS /COURSES (cont'd) 

6. Curriculum S Methods 1. Language Arts 

2* Social Studies 

3 • Mathematics 

H. Science 

7. Behavior Management 1. Managing Behavior. 

Practicum Experience 

The practicum consisted in many different activities, 
the majority of which centered upon classroom experience 
with supervision at Hillcrest. Other experiences included 
orientation, public school placement, residential place- 
ment, independent study, and evaluation days^ An overview 
of the practicum follows and the different experiences are 
described briefly. 

As noted earlier, the Psychoeducational Institute of 
Hillcrest Chili5ren's Center was chosen as the training centei 
for the American University program. In addition to train- 
ing and research components, the Psychoeducational Institute 
included pre-school, elementary school, and residential pro- 
grams for children with severe learning and behavioral 
difficulties. Although the majority of the children were 
emotionally disturbed, the pupils attending the school and 
residence were not a homogeneous group and presented a wide 
range of psychoeducational problems. 

For example, a survey revealed that (1) in terms of 
behavioral symptoms, 42% of the children were described as 
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hyperactive or hyperag.gressive V7hile "34% showed passive- 
aggressive or withdrawal patterns; (2) in terms of constitu- 
tional limitations, 28% had perceptual-motor handicaps while 
14% had a primary impulse disorder; (3) in terms of learning 
problems, 40% had reading difficulties and 25% had arith- 
metic difficulties. Children in the pre-school program, 
presented a combination of social, emotional and language 
handicaps • Hillcrest was located in the inner-city ghetto 
area of V7ashington, and U5% of the children in all 

school programs were inner-city residents. 

Although the training sequence varied somewhat from year 
to year, an orientation program was held in September, Train- 
ees were introduced to the facilities, programs and services 
available at Hillcrest Children's Center in both the Psycho- 
educational Institute (School and Residence) and the Clinical 
Institute (Department of Psychiatry, Children's Hospital of 
D.C.). They met with department heads, administrative of- 
ficials, school faculty, and center staff who provided them 
with a comprehensive overview of Center functions. Classroom 
assignments, seminar lists, and daily schedules were dis- 
tributed. Trainees worked out their independent study project 
reviewed records of the children in thei^ classes and planned 
their classroom assignments with master teachers, with whom 
they were to work daily throughout ' the year* 
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Beginning in late September and cont5.nuing into 
June> trainees spent approximately hours per week at 
Hillcrest teaching and tutoring, attending seminars and con- ^ 
ferences, and working on independent study projects. TS^o 
weekly schedules are presented below and these give some 
idea of the integration, if not saturation, of academic and 
practical experiences during the year. The schedules also 
indicate that classroom assignmentf> changed from first to 
second semester. (Four students in the classroom during the 
mornings of the first semester rotated to afternoons in the 
second semester; and vice versa.) 

Trainees were assigned in pairs to a particular class 
•and -master teacher. The. nature of the classroom changed 
gradually as the year progressed. Eai?ly experiences were 
primarily observation and participation under direction. 
Later experiences were more properly "student teaching" as 
trainees assumed more responsibility for curriculum and les- 
son plans, for individuals and for groups. In the Spring 
of each year, trainees assumed full responsibility for 
classroom planning and program for a specified period of 
time (usually four weeks). 

Throughout the year, trainees received supervision 
and feedback from staff members on the basis of both routine 
and request. Primary authority for overall supervision was 
that of the Director of Training. Because of on-the-job 
reality, however, the responsibility of daily supervision 
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*>ecame that of the four master teachers, supported by the 
principal and assistant principal of the school." (In the 
third yeai^ of the program, 1971-72, the assistant to the 
training director assumed much of the supervisory functions 
of the assistant principal.) These six staff members were 
closest to the classrooms and, in effect, to the trainees 
in the classroom. 

As was true of training sequence, the supervisory 
program varied from year to year. A review of schedules 
in 1971-72 provides a summary of practices for- that year. 
Classroom observations of each student by the principal oc- 
curred twice weekly, at a minimum. These observations usually 
lasted for 20 minutes (average .length of a lesson). Super- 
visory sessions by the principal were held once weekly; 
these may have included the student ^s master teacher as 
well. Observation of students by the assistant to the 
training director occurred once weekly for approximaxely 
^5 minutes. Feedback sessions were also scheduled once 
weekly; these may have included the student's co- U'ainee 
(in the same classroom). In addition to actual classroom 
observations, each student was video-taped in a teaching 
capacity for 2 0 minutes on a once-a-week basis. Video- 
tapes were readily available for students to review privately 
or with their master teachers and supervisors. Tapes were 
also reviewed in certain seminars, especially the behavior 
management and four curriculum seminars. 
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Public Schoo l Placement was considered a critical 
phase of the practicum year. Students' were usually placed 
with experienced, competent teachers who had classes of 
children with learning and behavioral problems. This ex- 
perience pr.ovided each student the opportunity to work with 
another master teacher, in a public school setting, and with 
children whose problems were usually less severe than those 
of the Hillcrest pupils. In addition to being supervised by 
the special teacher in the classroom, each trainee was ob- 
served for two hours per week by the director of training, 
by the principal of the Hillcrest school, or by the assistant 
to the training director. As noted in the training sequence 
section, during the first two years of the program the public 
school placement occurred in September and again in January 
for three weeks in each instance. The placements were in a 
special program in Montgomery County, Maryland. During the 
third year of the program, the public school placement oc- 
curred in January for a four w^ek period. Five of the 
students were in urban schools in Washington, D.C.; three 
of the students were in suburban Maryland schools. 

For most trainees in the first two years the residen- 
tial experience took place during the second semester. In 
the third year, this experience was scheduled in the first 
semester. Trainees were assigned to the residence unit of 
a one week period on a 2^ hour basis while they continued 
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classroom duties fro.n 9:00 to 3 : 00 • During this week students 
were under the super-vision of the Director of the Residence 
and worked as members of his staff. Conferences were held 
prior to this practicum for purposes of orientation and 
structuring of learning. A debriefing conference was held 
after the practicum for purposes of feedback and evaluation 
of the experience. This particular "living in" experience, 
though brief, proved to' be an intensive and rewarding aspect 
of the trainees' year. Playing, eating, and sleeping with 
the residence children and staff provided another dimension 
separate and distinct from classroom teaching. 

Independent study projects were worked on through- ^ 
out the year under the supervision of the training director . ' 
or, in the third year, the assistant to the director of 
training. Projects for the trainees were varied and selected 
on the basis of interest as an "elective." Projects have 
included: work in the nursery school with a 4-year old 
presenting severe language development problems; work as 
co-therapist in a therapy group for boys, ages 12-14 years; 
tutoring in a laboratory school for children with learning 
disabilities; in-depth diagnosis and tutoring of a child 
in a local public school; keeping a diary of events and re- 
actions during the year; recruitment of Black applicants 
for the training program. 



Evaluation Da ys V7ere scheduled at approximately six 
V7eek intervals: the Hillcrest school and residence were 
closed for an in-service training and evaluation day. Total 
Staff and the eight students met to assess individual and 
group progress of the students as well as to evaluate the 
entire training program. A basic assumption has been that 
student participation in program development is essential. 
Thus, these in-service days were also used to ijequest program 
modification. The evaluation days have been instrumental 
in producing significant improvements in the total program. 
They also provided students with the opportunity to evaluate 
an on-going educational program and to assume responsibility 
■for change. 



PROGRAM EVALUATION 



The Graduates 

One year after completion of the American University 
teacher training program each group of trainees is contacted 
and interviewed. Information regarding their present posi- 
tions is obtained and an attempt is made to keep a current 
record of each graduate from year to year. Two ^ of the three 
groups of graduates (16 individuals) have been interviewed 
to date. However, knowledge of present positions is avail- 
able on all three groups (.2^ individuals). Since completion, 
of the training program, 18 of the 2H trainees have continued 
to work in the field of education. Of these 18 trainees, 
5 are now working in administrative capacities. One of these 
is principal of a school for emotionally disturbed children, 
another is residence director of an institute for severely 
disturbed children, while a third is planning a residential 
treatment center. Of the remaining 2 trainees working as 
administrators , one is with a national office of education 
and the other is program director for an urban public school 
system. Thirteen past trainees are presently working as 
special education teachers, one of whom has already received 
a Ph.D. in Education. The types of handicapped children 
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served are delinquent, severely emotionally disturbed, learn- 
ing disabled, and behaviorally disordered. 

Three of the remaining 6 past trainees not currently 
working in educaticSn are continuing post-graduate studies 
at the doctoral level in education, clinical psychology, and 
.medicine. The remaining 3 trainees are involved in various 
other activities which essentially represent a year off from 
both studies and work. The following table summarizes the 
present positions of the 21 graduates to date. 



. TABLE 1 

PRESENT POSITIONS OF GRADUATES 
Academic Year 1972-73 



Present Positions ^ Number 

(N=21) 



A. Continuing as Students: 

Undergraduate 
■ Post Baccalaureate 
Post Master's 3 

B. Working As: 

Regular Classroom Teacher 

Special Education Teacher 13 
Supervisor or Administrator 5 

Other 3 

Total 21 



■Degree Obtained: 

Baccalaureate 
Master ' s 
Specialist 
Doctorate 
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Total 21 



COMPETENCY MEASURES 



The literature on training programs for teachers ' 
of the emotionally handicapped abounds in references to 
competencies. Since the list of "8 8 competencies" first 
appeared (Mackie, et al., 1957), numerous articles have 
been written expanding, refining, up-dating, and re-ranking 
the necessary quelities, characteristics, and competencies 
needed by teachers of emotionally handicapped children 
(Dorward, 1963; Hewett, 1966 ; Bulloch S t?helan, 1971). 

The implications of this literature for training 
programs have been clear: (1) a set of essential competen-- 
cies can be designated as training objectives; (2) 'given 
the objectives, a program of curriculum and experience can 
be devised to teach the competencies; (3) given the objec- 
tives and the program, evaluation can be carried out to 
determine the extent to which trainees learn and the program 
is effective. The experience of the authors has been that, 
although the plan of action is clear, following a three- 
phase blueprint of objectives — program — evaluation is not 
an easy matter* As is true of other areas of living, things 
are simpler said than done. 

28 
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The purpose of this paper is to report one attempt ^ 
at evaluation in a competency-based program. Data will be 
presented to show trainee changes over time as V7ell as their 
perceptions of the program. However, the major thrust of 
the paper is to describe both what was learned and the dif- 
ficulties encountered in completing research. No model for 
evaluation is being offered. Rather, the shortcomings and 
pitfalls encountered over a three year period are reported 
as part of the lessons learned. 

Instruments and Procedure 

A large battery of tests was administered the three 
groups of 8 students before and after each academic year 
(June). These tests included measures of competencies, 
attitudes toward education as well as interpersonal and 
personality characteristics. Only the competency measures 
are described and reported in this section. 

T^^o self-eyaluation instruments were employed in the 
study. The one consisted of the seven training goals and 
22 sub-goals vjhich students used to rate themselves on a 
9-point scale at five times^ during the year (June, September, 
December, February-March, and June). The second was a 137- 
itera Specialized Proficiency Questionnaire which was identical 
in most respects to the 157 item Teacher Competencies Check 
List of Mackie at al.,(1957). Students rates themselves 
using a 5-point scale at the beginning and end of the academic 
year. . }Vi> > . 
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The one supervisory rating scale adir.inistered V7ith 
some degree of regularity during the three years was adapted 
from a NIMH Project (Training of Teachers of Disturbed Pre- 
school Children, MH 1038 80) • The scale consisted of 42 
items to be rated by an individual trainee *s master teacher 
on a 7-point scale at t^^o times an the academic year (Octo- 
ber and June). Items were grouped on a rational basis into 
seven categories: involvement in program, orientation 
toward work with groups, orientation toward work with indi- 
viduals, relating to children, behavior management, and 
teaching techniques. Other efforts were made to obtain super 
visory ratings and staff evaluations of trainees but these 
were not consistent from year to year. Similarly, compre- 
hensive examinations of seminar content varied from year to 
year. 

A follow-up evaluation was completed for the first 
two groups of trainees one year after graduation from the 
program. This consisted in interviews of their administrator 
supervisor (i.e., usually principal), interviews of the gradu 
ates, as well as ratings of the program by graduates. 

Statistical Analysis 

Analysis of data in this study was accomplished by 
non-parametric statistical tests (Siegel, 1956). Non- 
parametric tests were chosen, among other reasons, because 
the statistical analysis has been sequential and the N^s 
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quite limited. Thus, many tables in.. the Results section 
report findings for the first year, for the second, for two 
years combined, for the third year, and for all three years 
combined. The Wilcoxin matched-pairs signed-ranks test was 
used for related samples as, for example, in the pre-post 
comparisons. The Mann-Whitney U test was used for inde- 
pendent samples as, for example, when different groups were 
compared. Only for the pre-post comparisons were one- 
tailed tests and associated probabilities used. 
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RESULTS 



1. Pre-Post Comparisons 

Self Evaluation of Skills . — Self ratings on the 
seven major tz^aining goals showed -a meaii increase in scores 
from pre- to post-testing. Table 2 summarizes the statisti- 
cal analysis of these data by reporting the probability 
levels derived from the Wilcoxin Test for the different 
analyses . 

TABLE 2 
SELF EVALUATION OF SKILLS 







1969-70 


1970-71 


1971-72 


2 Years 


3 Years 




Goals 


n=7 


n=8 


n=8 


K-15 


N=23 


1. 


System Awareness 


p. 05 


p. 05 


p. 01 


p. 005 


p.005 


2. 


Personal Sensitivity 


p.025 


p.025 


NS 


p. 005 


p.005 


3. 


Child Dsveloproent & 














Special Ed. Concepts 


p.025 


p. 01 


p. 005 


p.005 


p.005 

1 


H. 


Psychoeducational 








p. 005 


p.005 




Assessment 


p. 025 


p. 005 


p. 01 


5. 


Curriculum & Methods 


p. 01 


p. 005 


p. 01 


p. 005 


p.005 


6. 


Remedial Education 














£ Reading 


p.025 


p. 01 


p. 01 


p.005 


p.005 


7. 


Behavior Management 


p. 01 


p. 005 


p.025 


p. 005 


p.005 
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As can be seen in the Table, the increase in "perceived 
effectiveness" was significant in 20 of the 11 pre-post 
comparisons for the three groups of trainees considered sep- 
arately • The third year did not show a significant increase 
in Personal Sensitivity. VThen data are combined in the two- 
year and three-year analyses, significant increase is found 
for all seven training goals. As might be expected, these 
results clearly indicate that trainees perceived themselves 
as "more effective" in each competency at the end of the 
year than at the beginning. 

Specialized Proficiencies Questionnaire . — Results 
for the S.P.Q. are similar to those for self evaluation on 
training goals although fewer significant increases were found 
when the three sets of data were analyzed separately. Only 
19 of the 24» pre-post comparisons yield significant results 
aS' seen in Table 3. However, both the two-year and three- 
year comparisons show significant increase in all eight 
scales of the S.P.Q. 

Supervisory Ratings . Significant differences in 
supervisory ratings were not found in any of the three groups 
when separate analyses were made. Two of the scales (In- 
volvement and Teaching Techniques) were not found to be 
significant for either the 2 year or 3 year combined analyses. 
The Management scale was shown to have a significant increase 
for the 2 year combined analysis but not for the 3 year 



3H 



TABLE 3 

SPECIALIZED PROFICIENCIES QUESTIONNAIRE 



DCaJ.6 


1969-70 
n=7 , 


1970-71 
..n=8. , 


1971-72 

. . n=8 . 


2 Years 
. N=15 . . 


3 Years 
.N=23 


!• wricwing uiG v^rLLj.u 




D.05 


p. 005 


p. 005 


p.005 


2. CXirriculmri, Materials 
G nwXxiouo 


D.025 


p. 023 


p. 005 


p. 005 


p.005 


3. Testing S Psychoeduca- 
uLonai jJiagnosis 




n 01 


p.Ol 


p.005 


p.005 


li 'PiinHanno £ Thp'panptl't'ic 

Procedures 


p.Ol 


p. 025 


p.Ol 


p.Ol . 


p.005 


5. Teacher as a Profes- 
sional Team-^torker 


p. 01 


NS 


NS 


p.005 


p.005 


6. Parent S Public 
. 'Relations 


NS 


NS 


p. 025 


p.05 


p.005 


7. Teacher as a Person 


"p.025 


NS 


p. 005 


p.Ol 


p.005 


8. Personal Behavior 
Characteristics 


p. 025 


p. 05 


p.Ol 


p.005 


p.005 



^'Probabilities equal to or less than those reported 

combined data. The most encouraging results are seen in 
those areas related directly to working with children, i.e.. 
Group Orientation, Individual Orientation and Relating to 
Children. These 3 scales are shown to be significant for 
both the 2 year and 3 year combined analyses. These results 
are summarized in Table 4, which also reports the statistical 
results of a separate analysis for 14 trainees who had the 
same supervisor rate them pre- and post (cf . Section on Data 
Collection). ' 
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TABLE n '. 
MICHIGAN SUPERVISOR RATING SCALE 





Scale 


2 Years 


3 Years 


"14" 


1. 


Involvement 








2. 


Group Orientation 


p=.0294 


p=.027it 




3. 


Individual "Orientation 


p=.01tf3 


p=.Q110 


p=.0250 


4. 


Relating to Children 


p=.0132 


p=.0384 




5. 


Manag<=iment 


pr.0537 




p=.0281 


6. 


Teaching Techniques 








2.- 


''^Wilcoxin (exact) 
Posttest Comparisons 


1 tail. 







It is clear that trainees perceived themselves as 



more' effective at the end of the year than at the beginning. 
For example, the pre-post comparisons for both the Self- 
Evaluations and the S.P.Q. result in highly significant 
differences for all the goals and scales of the two instru- 
ments. Two additional questions were asked of these self 
ratings: (1) Did trainees perceive themselves as more ef- 
fective in certain skills rather than others? Specifically, 
were there differences among the goals at the end of train- 
ing? (2) Did the self reports indicate' any differences 
among the 3 groups of trainees at the end of training? 
Specifically, did the S.P.Q. reveal differences among the 
three groups? 
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Differences among the ^^oals' v — Table 5 presents the 
mean rating associated with each goal at the end of the year 
for each group of trainees and for all three groups of train- 
ees combined* As can be seen in the table, a definite rank 
order of the goals exists and this order is remarkably 
similar from year to year. Goals 1., 2., and 7 receive the 
highest ratings whereas Goals 3, and 6 receive the lowest 
(lower the score, higher the rating). Goal 5 appears to 
reside between these two clusters. 

TABLE 5 

MEAN RATINGS OF GOALS AT END OF THE YEAR 



Goals 1969-70 1970-71 1971-72 3 Years . 



1. 


System Awareness 


2.3 


3.3 


2.5 


2.72 


2. 


Personal Sensitivity 


2.0 


2.8 


2.5 


2.1J5 


3. 


Child Development 


2.7 


3.9 


3.75 . 


3.»f8 


^. 


Psychoeducational 
Assessment 


2.8 • 


it.O 


3.63 r- 

i 


3.51 


5. 


Curriculum and 
Methods 


2.9 


3.8 


2.75 ■ 


3.16 


6. 


Remedial Education 
and Reading 


3.8 


i^.l 


3.69 


3.87 


7. 


• 

Behavior Management 


2.1 


3.1^ 


2.13 


2.56 



Statistical analysis of these data was made by com- 
paring the ratings of each goal with every other goal. Again, 
the nonparametric V/ilcoxin Test was used. (Unfortunately,. 
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raw data for the firtjt group of trainees were not available 
for this analysis. Thus, data for only 16 trainees V7ere 
used.) Results of these multiple comparisons are consistent 
with conclusions reached by inspection of Table 5. Goals 
1, 2, and 7 do not differ significantly among themselves, 
■^y/hile each is significantly different from each of Goals 
3, 4, and 6 which also do not differ among themselves. (In 
the case of significance, p <.05, two-tailed test.) Goal 5 
is significantly different than only Goal 6 (p <.02).. 

In siuiunary, at the end of the academic year, trainees 
see themselves as more competent in the areas of system 
awareness (1), personal sensitivity (2), and behavior manage-^ 
ment (7) than with the areas of child development (3), 
assessment (U), and remediation (6). As for curriculum and 
methods (5), trainees rate themselves more competent in this 
area than in remediation (6). 

Differences among the trainees . — Table 6 reports the 
mean ratings associated with each scale of the S.P.Q. for 
each of the three groups of trainees, at the end of their 
respective years. Inspection of the mean scores reveal3 con- 
sistent differences among the three groups of trainees (high- 
er the score, higher the rating). The first group of trainees 
(1969-70) receives the highest rating on each S.P.Q. scale; 
the second group (1970-71) receives the lowest rating on each 
scale; and rating for the third group (1971-72) falls between 
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the other t'..'o. Differences among th.e three groups are 
statistically significant (Kruskal-V?allis one-v;ay analysis 
of variance). These results apparently stem from the highly 
significant differences between the first. and second group 
of trainees where p < .002 in 5 of 8 comparisons (Mann- 
\^hitney U Test). . 



TABLE 6 

SPECIALIZED PROFICIENCIES QUESTIONNAIRE 
MEAN DIFFERENCES: POST 





Scale 


1969-70 
n=7 


1970-71 
n=8 


1971-72 
n=8 


l-lann 
V/hitnQT 
p score 


Kruskal- 
Wallis 
p score 


1- 


Knowing the OiLld 


4.21 


3.30 


3.82 


.002 


.01 


2. 


Curriculiraj Materials 
& Methods 


4.27" 


3.46 


3.99 


.020 


.01 


3. 


Testing S Psycho- 
educational Diagnosis 




3.11 


3.67 


.001 


.01 




Guidance S Therapeutic 
Procedures 


4.12 


3.13 


3.65 


.002 


.01 


5. 


Teach^ as Professional 
Teanv-Worker 


4.14 


3.36 


3.70 


« 

.001 


.02 


6. 


Parent S Public Rela- 
tions 


3:88 


3.08 


3.50 


.010 ■ 


.05 


7. 


Teacher as a Person 


4.61 


3.74 


■ 4.18 


.002 


.02 


8. Personal Behaviox"^ 


4.55 


3.70 


4.04 


.010 


.05 



Characteristics 
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The higher the score on the-jS.P.Q., the "more confi- 
dent" the trainee feels about his ability on a particular 
item or scale. Thus, it is clear that the first group of 
trainees felt .more confident about their skills in all areas 
measured by the S.P.Q. than did the second group. These 
results pertain to the end of the year only. (At the begin- 
ning of the year, a significant difference between these two 
groups was found on only one scale — Personal Behavior Charac- 
teristics.) Reasons for this discrepancy are investigated in 
the Discussion section. 

3. Process Comparisons 

Sequential Evaluations . Another use of the Self- , 
Evaluation of Skills data can be seen in Figure if, which 
plots the mean scores of the 23 students for Goals 2 and 6 
across the five evaluation periods. This figure compares 
self reports of competency in personal sensitivity and cur- 
riculum and methods on a sequential basis. 

At pre-test for each year the trainees rated them- 
selves "above average" in the area of Personal Sensitivity, 
and "below average" for Curriculum S Methods skills. (It 
should be noted that the majority of trainees had no pre- 
vious educational courses or experience.) By December, the 
trainees saw themselves as strongly sensitive and with 
average curriculum and methods skills. At posttest Personal 
Sensitivity continued to rate higher than Curriculum 8 

47 ■ ■ . 
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Methods, although the greater improvement is seen in the 
latter (as shown by the steeper slope), and both are rated 
as being strong skills — above an '^acceptable" level of 
skill for a teacher .of emotionally handicapped children. 



Figure 4. Self Evaluation of Skills 
3 Years Combined — K Scores 




8 _ 

9_ 

Personal Sensitivity Curriculum S Methods 

In the case of other training goals, the sequential 
ratings, when plotted, correspond more to curriculum and 
methods than to personal sensitivity. Thus, during the first 
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3 months of the training program (Jyne-August) the trainees 
see themselves as making only a slight increase 'in their 
skills-level for the major training goals. This initial 
period after entering the training program is spent on the 
American University summer campus where the trainees are in- 
volved with three classroom courses. The actual practicum 
experience, involvement with the children and, consequently, 
the increases in skill-level begin after August when the 
trainees are brought into the school setting. The' exception 
to this general pattern is seen in the area of Personal . 
Sensitivity (as noted above). Also, Child Development £ 
Special Education Concepts and Psychoeducational Assessment, 
both. of which are areas of study in the summer school ses- 
sion, show a slight increase over the initial 3-month periods 

Self vs. system . — At the mid-point of the third 
tr'aining year (February, 1972), the Self Evaluation of Skills 
was used as a means for comparing supervisory ratings of the 
trainees with the trainees^ self evaluations in regard to 
the 7 major training goals. Each trainee was rated on each 
goal, the 22 sub- goals, and "general effectiveness" by his 
master teacher, by the Assistant to the Training Director 
(who worked in close contact with the trainees throughout 
the year) and by the seminar leader(s) whose course was 
centered on a specific training goal. 
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The self-system ratings were analyzed and. used in 
several ways. For example, individual graphs V7ere plotted 
for each trainee showing his ratings and the composite sys- 
tem ratings (mean scores of the supervisors) for the 7 
training goals and for "general effectiveness." The vari- 
ability among the supervisors was also shown on the graph. 
Individual feedback sessions were held with each trainee 
and his supervisors. In these sessions the t\<io sets of rat- 
ings were compared, discrepancies between trainee and system 
(as well as variability within the system) were discussed, 
and recommendations for the second, half of the year were 
made • 

As night be expected, ratings of some trainees were 
remarkably similar to those of the system while the ratings 
of others were quite discrepant. Similarly, some trainees 
elicited less variability among their supervisors while 
others provoked a good deal of disagreement • Finally, the 
7 goals were associated with varying degrees of variability. 
This issue of variability will be reported in greater detail 
in a following section on Data Collection. 

The group mean scores of the trainees were also 
compared with the combined mean scores of the system (i.e., 
the supervisors) and these are plotted in Figure 5. As 
can be seen in the figure, trainees tended to rate themselves 
more favorably on Goals 1, 2, 6, and 7 than did the system. 
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Figure 5: System Evaluation 8 Feedback training Goals. System Evalu 
of Group/Self Evaluation for Group. 
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Mean ratings for Goals 3, 4^ and 5 w.ere remarkably similar; 
while the system rating of "general effectiveness" v/as 
somev7hat higher than the trainees' rating on the Scune vari- 
able. (Statistical analysis of these data was not made. 
In all probability, observed mean differences are not sig- 
nificant because of the small number of ratings from the 
system — n=3 or 4 — and the variability in both sets of 
scores . ) 

4. Follow Up 

. Separate interviews are held vith trainees ajid with 
their supervisors one year after graduation for the pro- 
gram. (In most cases the supervisor is the principal of 
an elementary or special education school.) To date, the 
first two groups of graduates have been "followed" in June, 
'1971, and in June, 1972. 

Supervisory Interviews . — Supervisors of 5 of the 
first year graduates were contacted in 1971; however, super- 
visors of all 8 second year graduates were reached in 1972. 
They were asked to evaluate the former trainees* general 
"on the job functioning." Reported comments were very 
favorable. Phrases Such as "excellent, successful, con- 
cerned, creative, initiates follow-up and involvement, and 
relates well to both children and adults" were common. 
Two trainees, who graduated in the second class, were 



described as having had some inijriar difficulty , but both 
had shown tremendous growth and significant progress from 
the beginning of the year. 

Compared to other teachers trained in Special Educa- 
tion, 7 of 13 past trainees were rated by their present 
supervisors as "excellent, above average, or superior." 
Four were rated as "favorable." One graduate (from the 
second year) was rated as "better academically," but as having 
poor behavior management skills. This particular student 
was presently teaching a special education class of older 
.boys, having been trained in the area of younger children. 
One supervisor expressed that a former trainee "could be 
more systematic and dress more professional." 

When asked to express the "strengths" of the former 
trainees, supervisors were favorably impressed by their 
strong interest, concern and understanding of children's 
emotional needs . Several graduates had introduced new 
methods and activities into the school, which were obviously 
appreciated. Graduates were reported as being liked and 
respected by other staff members and they were able to ac- 
cept criticism and inaugurate change. Generally, they were 
"objective, consistent, cooperative, self-confident and not 
threatened by impulse-controlled children." 

"Weaknesses" were expressed in more speci.fic terms 
and often concerned personality variables. Three graduates 
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from the second year and one from the first were said to 
have V7caknesses v/hich V7ere related to their strengths. For 
example, comments made by the supervisors included: "expec- 
tations too high ... leads to disappointment; criticises 
self too harshly for unfavorable results; standards so high 
it is difficult for him to meet his own demands ; almost com- 
pulsive in pursuing action." Other areas of weakness were 
related to specific job functions and competencies,, such as 
in remediating perceptual difficulties and in psychoeduca- 
tional assessment. Only one supervisor expressed general 
dissatisfaction with a trainee's previous training and experi- 
ence. 

Perceptions of Graduates . —At the time of the follow- 
up interviews , graduates were asked to complete two rating 
scales both of v;hich pertained to the training goals. On one 
scale they addressed the question: "How relevant are the 
training goals for your present position?" On the other was 
the question: "How well did the A.U. /Hillcrest program train 
you in each of the areas?" The objective in both cases was 
to assess trainees' perceptions of the program and its goals 
one year after graduation. 

Table 7 presents the rank order of goals based on 
the graduate ratings in response to the two questions. As 
seen in the table, the same three goals elicited the high- 
est ratings. Goals 1, 2, and 7 were considered "most 



relevant" to present position as well as the areas in which 

•••• * 

the program had been "most effective." 



TABLE 7 

RANK ORDER OF GOALS ON THE BASIS OF 
RESPONSES TO THE TWO FOLLOV/-UP 
QUESTIONS 



Relevance 

2. Personal Sensitivity 
1. System Awareness 

7. Behavior Management 

3. Child Development/Special 

Education 
6. Remedial Education 

5 . Curriculum/Methods 

Psychoeducational Assess. 

2,1,7 significantly different 
than. 5,4. 



Program Effectiveness 

7. Behavior Management 
2. Personal Sensitivity 
1. System Awareness 



6. Remedial Education 
5 . Curriculum/Methods 
4. Psychoeducational Assess. 
3. Child Development/Special 
Education 

7,2,1 significantly different 
than 6,5,4,3. 



p < .01 



p < .05 



In the case of the question pertaining to relevance, 
statistical analysis of the ratings revealed that Goals 1, 
2, and 7 did not differ among them.selves but each were sig- 
nificantly different than Goals 4 and 5, which did not 
differ between themselves. Goals 3 and 6 fell mid-way 
between the two sets in terms of rank order but did not 
differ significantly from any other goal. 

In the case of the question pertaining to program 
effectiveness, statistical analysis revealed two different. 
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sets of goals. Thus, ratings for each of Goals 1, 2, and 7 
were significantly different than the ratings for 3, 4, 5, 
or 6. There vzere no dif fei-^ences among goals in each set. 

In siUTjnary, the first two groups of graduates per- 
ceived competency in system awareness (1)> personal sensiti- 
vity (2), and behavior management (7) as most relevant to 
their present positions compared to other goals. They con- 
sidered curriculum and methods (5) and (4) as least relevant. 
Child development (3) and remediation (6) were not dis- 
criminated from the other goals. . 

Regarding the question of problem ef f ectiveness > . 
graduates clearly differentiated between two sets of compe- 
tencies. In essence, they felt the program provided more 
effective training in those areas which they also considered 
most relevant: namely, system awareness (1), personal 
sensitivity (2), and behavior management (7). 

These data are remarkably similar to end-of-year 
results reported earlier: i.e., at posttest trainees per- 
ceived themselves as more competent in Goals 1, 2, and 7 than 
in 3, 4, and 5. Thus, a consistent finding emerges. Upon 
graduation trainees feel most competent in system awareness, 
personal sensitivity, and behavior management* One year 
after graduation they perceive these same competencies as 
most relevant to job functioning and consider the program 
most effective in training for these competencies. 
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Graduate Interviev7s . — In..,ftddition to the supervisory 
interviews and the rating scales, interviews were also con- 
ducted with each graduate at follow-up. Questions were asked 
to elicit their opinions of program strengths and limita- 
tions from the vantage point of one year after graduation. 

Content analysis of. the interviews revealed agreement 
on a number of strengths or assets of the program from both 
groups of graduates. Chief among these were the following: 
(1) The practicum experience offered by the program. (The 
fact that trainees had had experience with children appeared 
.to them to be a definite plus.) (2) The emphasis in the 
training center on understanding the dynamics of an individual 
child with problems together with the emphasis on designing • 
therapeutic solutions to problems. (3) The flexibility of 
the system primarily in terms of responding to trainees' 
needs by adapting the program; (h) The seminar in remedial 
education (reading). (This particular seminar was singled 
out by many graduates as an invaluable experience for them. ) 

A second "cluster" of assets of strengths was designate 
—perhaps less frequently than the first or in less glowing 
terms. Among these were the experience in the process group 
(seminar in group behavior) ; the opportunity and stimulus 
to learn more about adult relationships and to grow inter- 
personally; and the commitment and work of the Master 
Teachers. 

< t }o 
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There were three "v;eaknesses" or "limitations" 
in the program that V7ere clearly designated by the trainees 
one year after graduation. The first of these related to 
the lack of support and resources that graduates encountered 
in their. >work settings. They felt that the program had not 
alerted them to the problems or not prepared them to work 
without the support of a "therapeutic teaffi" in the educa- 
tional systems. A second criticism concerned the summer 
session on campus, which trainees considered too traditional 
or too academic. The third complaint was that still more 
emphasis should have been given to teaching techniques and 
content — both in classroom curriculum and in individual 
remediation. 

As with the strengths, there was a second cluster 
of weaknesses V7hich appeared less frequently or less em- 
phatically in the graduates' interviews.. Most of these 
focused on the need for "more" — more supervision and feed- 
back, more opportunities to assume responsibility, more 
pressure from staff to evaluate oneself, more experience 
with children on an individual basis, more understanding and 
skill in working with the group process presented by children 
in classrooms, more time for personal development. A 
criticism was also leveled at the problems inherent in the 
"system," at staff dynamics, as these hindered program ef- 
forts and trainees' learning. 




Discussion 

—————— ~— ..^ H 

Self-Evaluations V/hat is most obvious, perhaps, 
from these results is the abundance of data demonstrating 
significant improvement in competencies as perceived by 
trainees. This abundance of self- evaluative data contrasts 
sharply v/ith the dearth of evaluations by others — not only 
in terms of quantity but also in terms of a proportional 
RUJijber of significant changes. Still, our feeling is that 
the self-evaluation data were indeed worthwhile and provided 
valuable feedback to the system. For example, trainees' 
views of their competencies often became a sensitive barometer 
of their feelings about themselves and where they were at a 
particular point in the training sequence. Thus, a "February 
slump observed in one year could be anticipated in the next." 

Differences between groups of trainees were also 
noted as, for example, those summarized in Table 4-. The 
first group of trainees (academic 1969-70) definitely saw 
themselves as more competent than the second group (academic 
1970-71) at the end of the training program. Several inter- 
pretations of this result could be made. One is that/the 
trainees* evaluations mirrored staff enthusiasm. Thus, the 
sparkle of a program in its first year may have been tar- 
nished somewhat by the working through of reality issues in 
the second year. And trainees' self-evaluations for the two 
years corresponded to the decrease in enthusiasm and the 
increase in reality. , 
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Trainees' ratings of their competencies could also 
be taken at face value. Under these conditions they revealed 
differential attainnient of goals and, in so doing, described 
program strengths and weaknesses or> at the least, program 
emphasis. For example, upon graduation trainees saw them- 
selves as more competent in the areas of personal sensitivity, 
system awareness, and behavior management than they did in 
psychoeducational assessment, remedial reading, and child 
development. The three goals or competency areas of personal 
sensitivity, system awareness, and behavior management were 
again spotlighted on foilow-up of the graduates. These 
three \>7ere seen as ^^jv.oet relevant" to on-the-job functioning. 
The program was seen as "most effective" in training in these 
three areas.. 

A final point in favor of the self evaluations con- ' 
cerns their usefulness in comparing the perceptions of traine 
and staff (cf. "self vs. system" analysis). Use of the self 
ratings in foriiial sessions with individual trainees and staff 
became a valuable tool in the evaluation and feedback cycle. 
Designating a specific time of year (usually March) to 
compare the perceptions of self and other — to focus on assess 
ment— had the advantage of keeping everyone honest and not 
allowing feedback to attenuate during the year or, worse 
yet, to be forgotten until the end of the year. 

• 6'. 
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In suiTjnary, our feeling has^-been that self ratings 
are extremely useful in the ovsrall scheme of research. Al- 
though they are certainly not the total answer in evaluating 
a competency-based program and its trainees, they provide a 
great deal of information. They have told us much about 
our program as well as about our trainees and graduates. 

Research Design and Data Collection . — Despite the 
utility of the self ratings, it is clear that the research 
design was inadequate in planning for independent evaluations 
within the training' system. It was also beset by problems 
in data collection and in the variability among the ratings 
\obtained. The attempt was certainly made to include super- 
visory ratings (i.e., Master Teachers) as a pre-post measure 
in this research. However, it was impossible to ensure that 
the supervisor who rated a particular trainee at the begin- 
ning of the ysar would also rate that person at the end of 
the year. In fact, only 14 of the total 24 trainees from 
all three years were rated by the same supervisor twice (cf . 
Table 4 in Results section). 

Collection of data from the same rater may have been 
better planned for and more vigorously pursued. Still, 
there remain the inevitable program changes and supervisor 
transfers which complicate data collection. There also 
remains the problem of variability among the raters: e.g., 
differing "frames of reference" for items on a rating scale, 

9^- ; • 0... 
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different expectations for trainees in terms of competent" 

vs "highly competent," differing behaviors revealed by 

trainees in the presence of one rater as opposed to another • 

Also, when supervisor and trainee are intimately Involved 

in a relationship — such as that of master and student teacher 

working V7ith "disturbing children" in the same classroom-- 

the effects of this interaction on objective ratings are 

probably legion • 

The difficulty of collection of data V7as also in- 

X . creased due to the lack of a "testing environment" in which 

the supervisors rated the trainees , in addition to the 

intrinsic resistance of teachers toward evaluation. Most 

master teachers had difficulty finding the time in their 

daily schedules to sit down and properly rate the trainees. 

The rating scale became a burden as they felt unable to 

disregard other responsibilities in order to complete the 

ratings. The ratings were made at the beginning and end of 

the year — busy times in any teacher^ s life. 

Even in the area of self evaluations ^ difficulties 

arose. In the first year of the training program (where 

n=7), data were lost because of a Cdtiscientious Test 

Objector. V/hat effect his refusal to complete "required 

testing" had on responses of other trainees is not clear. 

However, his refusal could hardly have been missed and the 

effect was most probably experienced at some level as 

•I 
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resentment toward the tests . (A more frequent occurrence 
than the Conscientious Test Ob3ector may well have been the 
passive-aggressive test taker v7ho, wittingly or unwittingly 
provided data of dubious value.) V/hatever the reaction of 
the training director an^/or research staff to the C.T.D., 
one has the decision to malce of drawing a disjunctive line 
(either take the tests or leave the program) or of granting 
"special privileges" to one m'fember of ,a group. 

The descriptive data obtained from follow-up interviews 
of supervisors and administrators were encouraging in that 
many positive responses were made regarding trainees after 
their first year on-the-job. Nevertheless > testimonials are 
fairly easily garnered but "harder data" are just that — 
harder to come by. Inclusion of a rating scale v/ould not 
by itself have ensured the hard data; however, it could have 
been a step in the right direction > 'p'»rticularly if those 
outside the system were rauing the goals and objectives of 
the program. 

Another apparent weakness of the follow-up data from 
"independent sources" was that only seven of tx>7elve trainees 
were, in fact, rated by independent sources. Six of sixteen 
were employed at the teaching center during their first year 
on the job and thus evaluated by staff personnel. Such an 
occurrence would be unlikely in a more traditional university 
program. However, as more training programs become housed 



in clinical centers as opposed to the university classroom, 
such an occurrence may become more frequent. The result 
being fewer independent observations from the consumer. 

A final comment on these results concerns the issue 
of teacher competency v£ effectiveness. As noted by Scheuer 
(1971) the two terms are not synonymous nor does the first 
necessarily lead to the second* One of the best times to 
obtain measures of trainee effectiveness is during the on- 
the-job situation of the first year imen much behavior is still 
contingent upon the training program. Again, the responses 
of principals in this study was encouraging as indeed they 
felt many trainees were highly effective in their positions. 
Still no measures of teacher-pupil rapport, or of classroom 
organization or of children's learning and development were 
attempted- Indeed, the thought of effectiveness measures 
elicits a thought or two about control group (as in "effective,, 
compared to whom?"). An expe.-^imental control design was not 
used in this study though it obviously made sense to do so 
and even though one trainee had asked some years ago, "who's 
my control group?" 

Im.plications 

A clear guideline for the future is to reduce the 
variability in both self evaluations and supervisory ratings. 
Program objectives (competencies-to-be-learned) must be ex- 
pressed more precisely and defined more operationally. Greater 
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efforts at supervisor training in rating behavior must also 
be made» In the case of multiple raters, pre- training ses- 
sions to establish common frames of reference for variables 
and common expectations for trainees should be routine. 
Scheduling of time for master teachers to complete ratings 
free of other duties is necessary y.e± not always a simple 
matter in the v;orkday world V7ith emotionally disturbed children 
Use of video-taped vignettes could be used to reduce variance 
provided the sequences are held as constant as possible (e.g., 
teaching a class lesson, tutoring an individual, crisis 
intervention) and identical groups of raters are used to 
evaluate the sequence. 

In ooncluding, it seems advisable to set down a number 
of principles which have evolved during the course of our 
three years in program development. These pertain, of course, 
to the evaluation process only. 

1. The first principle has been that program and 
program development take precedence over research design. 
\^hat this means in practice is that many changes will be made 
from year to year. Some of these have little effect on evalua- 
tion procedures (e.g., modifications in the selection process). 
Others have direct bearing (e.g., change in supervisor). Also 
a great deal of "research activity" is devoted to immediate 
concerns of the staff and trainees (e.g., evaluation of ad- 
ministrative roles and functions; input into the observations; 
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evaluation of feedback process between staff and trainees; 
assessment and refinement of the selection process). 

2. The second principle has been that people take 
precedence over research. People in this instance refers to 
trainees as well as staff and pupils. Perhaps the best ex- 
ample of this principle occurred in the first year of the 
program when our Conscientious Test Objector was encountered. 
In this case a decision was comfortably made not to exclude 
the C.T.O. from the program: i.e., a promising trainee was 
"saved" at the expense of research data lost. 

3. The third principle has been that research and 
evaluation are not limited to tests, ratings, measurement 
and statistical analysis. In fact, these tools and products 
may lull a program to sleep through periods of needed self 
analysis. Program evaluation to be meaningful usually demands 
more pain than mere testing and rating. ^ Confrontation within 
the system (e.g., staff with trainees, trainees with staff) 
leading to catharsis and change is often required. Our 
vehicle for this type of research has been the Evaluation Day. 

4. The final principle has been that research, like . 
learning, doesn't end with graduation. We are av/are that more 
extensive use of independent evaluators could be made. For 
example, administrators and supervisors of trainees in their 
first placement should provide valuable feedback to the 
program as regards both competency and effectiveness. However 

I 0/ 
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the graduates, after a year or tvjo ay;ay from the training 
institution, are a significant source of data about both 
themselves and the program. V7ith experience and perspective 
their evaluations are no longer those of trainees rating self* 
Rather, they are more like interested, and independent, col- 
leagues rating us. 
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PERSONALITY MEASURES 



In addition to the competency measures administered 
pre and post, a number of attitudinal and personality tests 
^were used. The reasons for inclusion af non-competency mea- 
sures were threefold: (1) to determine program effects 
on personality variables of trainees; (2) to learn more about 
the trainees themselves as well as to compare the three 
groups; (3) to learn something a tout the measures used and 
their utility for further research. 

Instruments and Procedure 

A large battery of tests was administered each 
group of 8 students before and after each academic year 
(June)* These tests included measures of competencies, 
attitudes toward education as well as interpersonal and 
personality characteristics* All but the competency measures 
are described and reported in this paper. 

Instruments considered to be measures of attitudes 
toward education included: the Minnesota Teacher Attitude 
Inventory (Cook, Leeds S Callis, 1951), which is designed 
to sample opinions about teacher-pupil relations; the VAL-ED, 
a FIRO scale (Schutz, 1962), which related to personal atti- 
tudes and values toward various aspects of education; the 

61 
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Teacher Practices Qusstionnaire .^(SQrenson, Husek, S Yu, 
1963)5 which is a method of studying the teacher's (trainee's) 
indicated actions under certain classroom problem situa- 
tions; and the Problem Behavior Analysis in which specific 
classroom behaviors are described and the trainee rates each 
item according to what he anticipates will be the frequency 
of occurrences of that behavior in his classroom as well 
as his personal reaction to the behavior. 

The three instruments employed as measures of Inter- 
personal characteristics were the FIRO-F and the FIRO-B 
(Schutz, 1967) which explore the typical ways in which the 
student feels about people and interacts with people (res- , 
pectively), and the Communication of Feeling Inquiry (COFI) 
(Wallen, 1968)3 in which the rater is asked to distinguish 
between sentences that convey feelings by describing specific- 
ally what the speaker is feeling, and sentences that convey 
feelings but do not describe what the speaker feels. 

Personality characteristics were measured by the 
Adjective Check List (Gough 8 Heilbrun, 1965) and the Cali- 
fornia Psychological Inventory (Gough, 1956). 

Statistical Analysis 

As was the case with the competency measures, analysis 
of these data was made by non-parametric statistical tests 
(Siegel, 1956). The Wilcoxin matched-pairs signed-ranks 



test v;as used for related samples; the Mann-Uhitney U test 
was used for independent samples . All tests of significance 
were two-tailed. 



RESULTS 



1> Attitudes Toward Education 

Minnesota Teacher Attitude Inventory > — The M. T.A.I* 
consists of 150 items for which the rater is asked to state 
levels of agreement or disagreement about teacher-pupil 
relations. One score is derived from the 150 items. No 
significant changes in the trainees^ opinions were found 
comparing beginning and end-of-the-year ratings. 

VALUED • — Significant results v;ith the VAL-ED have 
been limited to the first group of trainees (1969-1970 gradu- 
ates). At post- test this group showed a significant change 
on the *'Mind" scale 5 indicating that the school should con- 
cern itself less with developing the mind of the student 
and more with developing his v7hole personality. The first 
year trainees also showed a significant change on the scale 
regarding Teacher-Child: Control. At end of the year 
trainees felt less strongly that "the teacher should regulate 
completely classroom lessons and activities." These results 
pertained only to the first group, however. Similar results 
were not found for subsequent groups nor in either the 2- 
year or 3-year analyses. 
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Teacher Practices Questionnaire > — Although neither 
of the first two years showed significant ch=inge on the five 
T.P.Q* scales, an analysis of the two years combined pro- 
duced a significant increase (p < •05) in the attitude that 
the teacher^s role as a counselor is "appropriate"; i»e», 
"he seeks basic underlying cause of behavior and helps the 
student to see various courses of action and to think inde- 
pendently*" 

Trainees graduating in the third year (1971-1972) 
did not show similar results, although, interestingly, it 
was found that this group rated higher on the Advice- 
InfoxTTiation Giver scale, expressing that this role is less 
appropriate; i.e», "the teacher should not use rxperiences ' 
and knowledge to advise a course of action, solve the 
problem and make the decision for the student*" 

Problem Behavior Analysis * — Combined analysis of 
the first two training years indicated that trainees felt 
less personal disturbance from overt, aggressive behavior 
of children* This finding was also obtained in the third 
training year and in analysis of the three years combined 
(see Table 8). 

It was also found that the third group of trainees 
expressed that, at the completion of training, they were 
less disturbed by a child's oppositional behavior and devia- 
tions in social behavior* 
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TABLE. "8- 

ATTITUDES TOWARD EDUCATION PROBLEM 3EK/^VI0R ANALYSIS 



1969-70 1970-71 1971-72 2 year 3 year 
Scale n=8 n=7 n=8- n-15''« n=15" 



1. 


Oppositional Behavior 
(reaction) 


p 


• 01 

• 


2. 


Overt, Aggressive Be- 
havior (reaction) 


P 


• 02 p .01 p .01 


3. 


Deviations in Social 
Behavior (reaction) 


P 


.05 


4. 


Oppositional Behavior 
(frequency) 






5. 


Overt, Aggressive Be- 
havior (frequency) 






6. 


Deviations in Social 
Behav^.cr (frequency) 




• 



^''All changes are in the direction of being less dis:- 
turbed about the particular behavior. 



2. Interpersonal Scales 

FIRO-F . — An interesting finding from the FIRO-F 
scale appears in the analysis for the three years combined* 
It seems that after completion of the training program, 
trainees see others as less "lovable"; others express less 
affection. It is also found that the trainees see others as 
less competent, they express less trust for others. 

FIRO-B . — No significant changes were found on any 
scale of the FIRO-B scale. 
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Comjnunication of Feeling; Inquiry , — The COFI consis- 
tently showed an increase of skill in conununication of feel- 
ings upon the completion of training for the first year's 
analysis, the first two years combined and the three years 
combined. 

3. ' Personality Characterisuics 

California Psychological Inventory , --Results derived 
from the C.P.I, for the first year trainees (1969-1970) 
show a decrease in ratings from pre- to post-testing periods. 
Significant differences were found for the scales of Sense 
of V/ell-Being^ Responsibility, Socialization, Self-Control , 
and Good Impression (p < .05, in all cases). 

Analysis for the second year trainees (1970-1971) 
showed that this group generally rated higher at post- test. 
Significant results were found for the scales of Sociability 
(p < .02), Tolerance (p < .05), Achievement via Confornance 
(p < .05), and Achievement via Independence (p < .05 for 
the second year and for the two years combined). 

Looking at the data for these two years, what first 
appears to be contradictory results is^ in actuality, a 
reasonable outcome of the program. The first year began 
at pretest with relatively high ratings and decreased to 
lower scores at posttest. The second year scored relatively 
lower at protest and higher at posttest. Statistical com- 
parison of the C.P.I, data for first and second year 
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trainees (Mann-Whitney U Test) demon-strated a regression 
toward the mean. Thus, at pretest the first year trainees 
score significantly higher than the second year trainees on 
seven separate scales; V7hereas, at posttest they score higher 
on only two scales (Sociability and Social Presence). The 
third year trainees did not show significant changes from 
pre- to posttesting. 

The combined C.P.I, profile for the three groups 
of trainees at graduation is presented in Figure 6. A look 
at the combined mean scores for the three training years shows 
a general trend of the trainees to rate themselves lowest on 
scales regarding social norms and values. The lowest of 
these (Responsibility, Good Impression, and Socialization) 
identify behaviors which are somewhat cautious, defensive, 
and disbelieving of the structuring of social values. The 
characteristic manner just described may also be seen by 
noting the high scores found in Flexibility and Psychological- 
Mindedness. Here, the trainees' broad attitudes toward life 
express themselves as being verbally fluent toward the inner 
needs, motives and experiences of others, while being re- 
bellious toward rules, restrictions and constraints. The 
trainees might be described as opinionated, with a high 
degree of adaptability of thinking and social behavior. 

Scales regarding feelings of interpersonal and iatra- 
personal adequacy show the highest ratings for Self-Acceptance 
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and Social Presence. These scales" 'express personal and 
social Tuanner^isms described as intelligence, enthusiasri, 
outspokenness , demanding , imaginative 5 and spontaneous . 
Finally, high scores on Achievement via Independence com- 
pletes the groups' profile as mature individuals with strong 
interest and motivation which facilitate achievement in set- 
tings where autonomy and independency are positive behaviors • 

Adjective Check List — Self Ratings > — There were very 
few significant pre-post changes on the Adjective Check List 
for each of the three groups of trainees. Of the 24- scales 5 
only two showed change for the first year, 3 for the second, 
and 1 for the third. The one consistent change over time 
was associated with the "number of favorable adjectives 
checked" which increased significantly for 2 years combined 
(p < .01) and for 3 years combined <p < .05). 

Comparison of the first two years of trainees on 
self-ratings yielded results contrary to those obtained with 
the California Psychological Inventory. Thus, at pretesting 
only 2 scales showed a significant difference between the two 
groups <p < .05, Mann-Whitney U Test, two-tailed) whereas at 
posttest 5 scales showed a significant difference. First 
year trainees rated themselves higher in self confidence, 
lability and need for achievement while checking more 
adjectives and more favorable adjectives. 
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In contrast to the C.P.I.-.findings , the combined 
A.C.L. profile for the throe groups is remarScably flat with 
all scales clustering around the mean. 

Ideal ratings -— None of the Adjective Check List 
scales showed significant" changes for two-years or three- 
years combined analysis. There were significant changes, 
however, associated with each year: 1 for the first and ' 
third years J 6 for the second year. Findings for this par- 
ticular year suggested real changes in the "ideal" for this 
group of trainees in that they wanted to be more self- 
confident, have more of a need for heterosexuality , autonomy 
•and exhibition, and have less of a need for abasement. 

On two oif the scales just mentioned, the second year 
trainees also revealed significant changes in self -ideal 
discrepancy scores from pre- to post-testing. The difference 
between self and ideal ratings for self-confidence and for 
heterosexuality increased as it also did for the lability 
scores. A specific example of the change in discrepancy 
score can be given for the 'self-confidence scores: mean 
self rating scores decreased from U7.U to UU.8 (nonsignificant 
in itself) while mean ideal ratings increased from 52.6 to 
61.3 (significant at p < .05). 
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Discussion •- - 

Program .— One of the main reasons for including 
attitudinal and personality measures in the test battery 
was to learn if our teacher training program had an effect 
on trainaes' personality. Conditions inherent in this 
particular prograni contained both advantages and disadvantages 
for such a study. Among the former was the length of the 
program— three years— providing as it did three separate 
samples of trainees or essentially tv70 replications of the 
study. Among the disadvantages was the nature of the program 
itself— new, experimental, certain to change to some degree 

from year to year. 

Because of the experimental nature of the program 
(as well as the small number of subjects in each group) the 
failure to find identical changes in each of the three groups 
may surprise no one. Although consistent findings did not 
occur from year to year, a pattern of change did emerge for 
all three years combined.- To sumjnarize results for all three 
groups, when posttest measures are compared to pretest, 
trainees (1) reported that they felt less disturbed by 
overtly aggressive behavior of children (P.B.A.); (2) in- 
creased their skill in decoding affective statements (CO. F.I 
(3) tended to see others as .less lovable and less competent 
(FIRO-F); and (U) checked a greater number of "favorable 
adjectives" when rating themselves (A.C.L.). 
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On the surfa-ie, the reports .pf four "significar.t" 
findings out of the many possible may not seem too striking. 
However, the specifics of those findings deserves some 
comment. The first two, for example, are considered essen- 
tial for helping children cope with their feelings (Long, 
et al., 1969). Aprocess of desensitization to the aggres- 
sxon of rebellious, acting out children must occur before a 
teacher or therapist or adult is in a position to help, to 
control, to work through that aggression. The ability to 
discriminate and to identify feelings— whether for receiving 
or for sending— is also a prerequisite to help those child- 
ren who are so confused and disturbed by their own feelings. 
Thus, i:hese reports of change in attitudinal and inter- 
personal behavibr are considered highly significant from 
the point of view of program objectives. 

The third finding--the tendency. to see others as 
"less lovable," "less competent"— is seen as a natural out- 
come of a year's intensive contact with children who indeed 
perceive themselves in similar terms, who question their 
own lovableness and competency, their worth and adequacy. 
Perhaps this finding represents for the trainees the notion 
of an "ideal tarnished"— "others are simply not as lovable 
or as competent as I would like them to be." In the context 
of the trainees' experience, it does seam to represent at 
least a move away from the ideal and toward the real. 
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Despite the inherent pessimism of this finding, the one 
clear-cut result frora the Adjective Check List provides a 
certain degree of optiirdsm* For the trainees use a greater 
number of "favorable adjectives" in describing themselves 
at end-of-year than at the beginning. Thus, experience in 
the program may have led to a dampening of ideals about 
others but the same experience resulted in a m.ore favorable 
perception of self. 

Trainees > — Another reason for including personality 
measures in the test battery vras to learn more about the 
trainees in the program. Although the group profile on the 
A.C.L. was neither descriptive or unique, that for the C.F.I 
appeared richer in the information provided. (Indeed a 
personality description could be written from the profile.) 
The similarity in C.P.I, profiles for all 3 groups— at' 
least at posttest — was striking. Although individuals 
certainly varied from the group profile, one could sense the 
group personality. Comparison of this group profile to pro- 
files for other educational and occupational samples (Gough, 
1956) indicated h definite sL-nilarity betx^een our trainees 
and psychology or social work graduate students. 

Comparisons among the three groups of trainees 
were also made and these proved informative, not only in 
discriminating among the groups but in showing indirectly 
changes in the program. For example, results of the 
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compefency measures indicated that the fir-st year rated 
themselves signi.Cicantly better than the second year on a 
nunber of competencies. Differences between the first and 
second groups of trainees on personality measures were found 
for both the C.P.I, and the A.C.L. One aspect of these 
findings suggested the greater poise, self assurance, and 
self confidence of the first group compared to the second. 
Also, the second year trainees apparently demonstrated mere 
dissatisfaction with its "ideal" as indicated by the greater 
frequency of change in ideal ratings from pre- to post-test. 

Thus, the second group of trainees appeares less 
sure of themselves, more dissatisfied, in greater turmoil, 
more introspective. The .differences between the groups are 
well documented. However, several questions remain un- 
answered. Were these characteristics actually intrinsic to 
this particular group? Did these findings result from pro- 
gram parameters? Was there ar. interaction between trainees 
and program- in-its-secor.d-year? One fact is known: during 
the seco3;d year fairly dramatic changes in administrative 
and supervisory practices were effected. These changes were 
the direct result of trainees' input on "evaluation days" 
and of their action. There is some evidence then, to 
conclude that personality differences (and competency ratings) 
of the second year group were related Co the program. The 
"Relationship is complicated. For the differences may have 
been both cause and effect of program changes. 
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Instruments .- -As for the measures themselves soir.e 
•were rarely associated v/ith significant findings; vjhile 
others were quite productive in statistical outconie. The 
former was particularly true of attitudinal measures (M.T.A.I. 
VAL-ED). One might be tempted to say these measures are 
not sensitive to change. . Our feeling at the moment is that 
a longer period than one year may be needed , to demonstrate 
attitudinal change. Specifically, a one year follow up of 
graduates may be necessary to detect changes in attitudes, 
particularly on the VAL-ED. Given a year in*another setting, 
one admittedly more real- life, might allow for more oppor- 
tunity to observe, experience, re- think and reshape attitudes. 
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